Velocity field interaction for free space gesture interface and control

ABSTRACT

The technology disclosed relates to automatically interpreting motion of a control object in a three dimensional (3D) sensor space by sensing a movement of the control object in the (3D) sensor space, interpreting movement of the control object, and presenting the interpreted movement as a path on a display. The path may be displayed once the speed of the movement exceeds a pre-determined threshold measured in cm per second. Once the path is displayed, the technology duplicates a display object that intersects the path on the display. In some implementations, the control object may be a device, a hand, or a portion of a hand (such as a finger).

PRIORITY DATA

This application is a continuation of U.S. application Ser. No.17/379,915, entitled “VELOCITY FIELD INTERACTION FOR FREE SPACE GESTUREINTERFACE AND CONTROL”, filed Jul. 19, 2021 (Attorney Docket No. ULTI1008-6), which is a continuation of U.S. application Ser. No.16/860,024, entitled “VELOCITY FIELD INTERACTION FOR FREE SPACE GESTUREINTERFACE AND CONTROL”, filed Apr. 27, 2020 (Attorney Docket No. ULTI1008-5), which is a continuation of U.S. application Ser. No.16/570,914, entitled “VELOCITY FIELD INTERACTION FOR FREE SPACE GESTUREINTERFACE AND CONTROL”, filed Sep. 13, 2019 (Attorney Docket No. ULTI1008-4), which is a continuation of U.S. application Ser. No.16/213,952, entitled “VELOCITY FIELD INTERACTION FOR FREE SPACE GESTUREINTERFACE AND CONTROL”, filed Dec. 7, 2018 (Attorney Docket No. ULTI1008-3), which is a continuation of U.S. application Ser. No.14/516,493, entitled “VELOCITY FIELD INTERACTION FOR FREE SPACE GESTUREINTERFACE AND CONTROL”, filed Oct. 16, 2014 (Attorney Docket No. LEAP1008-2/LPM-1008US), now U.S. Pat. No. 10,152,136, issued Dec. 11, 2018,which claims the benefit of U.S. Provisional Patent Application No.61/891,880, entitled, “VELOCITY FIELD INTERACTION FOR FREE SPACE GESTUREINTERFACE AND CONTROL,” filed on Oct. 16, 2013 (Attorney Docket No. LEAP1008-1/1009APR). The priority applications are hereby incorporated byreference for all purposes.

INCORPORATIONS

Materials incorporated by reference in this filing include thefollowing:

“INTERACTIVE TRAINING RECOGNITION OF FREE-SPACE GESTURES FOR INTERFACEAND CONTROL,” U.S. Provisional Application No. 61/872,538, filed Aug.30, 2013 (Attorney Docket No. LPM-013GPR/7312701007),

“Methods and systems for identifying position and shape of objects inthree-dimensional space,” U.S. Provisional Application No. 61/587,554,filed Jan. 17, 2012, (Attorney Docket No. PA5663PRV),

“SYSTEMS AND METHODS FOR CAPTURING MOTION IN THREE-DIMENSIONAL SPACE,”U.S. Provisional Application No. 61/724,091, filed Nov. 8, 2012,(Attorney Docket No. LPM-001PR2/7312201010),

“NON-TACTILE INTERFACE SYSTEMS AND METHODS”, U.S. ProvisionalApplication No. 61/816,487, filed Apr. 26, 2013 (Attorney Docket No.LPM-028PR/7313971001),

“DYNAMIC USER INTERACTIONS FOR DISPLAY CONTROL,” U.S. ProvisionalApplication No. 61/752,725, filed Jan. 15, 2013, (Attorney Docket No.LPM-013APR/7312701001),

“MOTION CAPTURE USING CROSS-SECTIONS OF AN OBJECT,” U.S. Provisionalapplication Ser. No. 13/414,485, filed Mar. 7, 2012, (Attorney DocketNo. LPM-001/7312202001), and

“SYSTEM AND METHODS FOR CAPTURING MOTION IN THREE-DIMENSIONAL SPACE,”U.S. Provisional application Ser. No. 13/742,953, filed Jan. 16, 2013,(Attorney Docket No. LPM-001CP2/7312204002).

BACKGROUND

The subject matter discussed in this section should not be assumed to beprior art merely as a result of its mention in this section. Similarly,a problem mentioned in this section or associated with the subjectmatter provided as background should not be assumed to have beenpreviously recognized in the prior art. The subject matter in thissection merely represents different approaches, which in and ofthemselves can also correspond to implementations of the claimedtechnology.

Users interact with a touch-screen user interface of a device with touchgestures. The device detects one or more touch events (e.g., tap, swipe,pinch-in, rotate, etc.) when the user performs a touch gesture on thetouch screen using fingertips or other pointing devices. The deviceinterprets the user's detected touch events. Detection andinterpretation of a touch gesture can be well defined by the locationand movement of the physical contact (or close proximity) between theuser's fingertip(s) and the touch screen.

Interpreting a user's gestures in a three dimensional (3D) free spaceplaced in front of a device is challenging as often there is no clearindication whether the user's gesture in the 3D free space engages thedevice. It is also challenging in determining a particular portion or aparticular hierarchical level of a user interface that the user isinteracting with using gestures in the 3D free space.

SUMMARY

The technology disclosed relates to automatically interpreting a gestureof a control object in a three dimensional sensor space by sensing amovement of the control object in the three dimensional sensor space,sensing orientation of the control object, defining a control planetangential to a surface of the control object and interpreting thegesture based on whether the movement of the control object is morenormal to the control plane or more parallel to the control plane.

The technology disclosed also relates to automatically interpreting agesture of a control object in a three dimensional sensor space relativeto a flow depicted in a display by sensing a movement of the controlobject in the three dimensional sensor space, sensing orientation of thecontrol object, defining a control plane tangential to a surface of thecontrol object and interpreting the gesture based on whether the controlplane and the movement of the control object are more normal or moreparallel to the flow depicted in the display.

The technology disclosed further relates to navigating a multi-layerpresentation tree using gestures of a control object in a threedimensional sensor space by distinguishing between the control objectand a sub-object of the control object by sensing a movement of thecontrol object in the three dimensional sensor space, interpreting themovement of the control object as scrolling through a particular levelof the multi-layer presentation tree, sensing a movement of thesub-object in the three dimensional sensor space and interpreting themovement of the sub-object as selecting a different level in themulti-layer presentation tree and subsequently interpreting the movementof the control object as scrolling through the different level of themulti-layer presentation tree.

The technology disclosed also relates to navigating a multi-layerpresentation tree using gestures of a control object in a threedimensional sensor space by distinguishing between the control objectand one or more sub-objects of the control object by sensing a movementof the control object in the three dimensional sensor space,interpreting the movement of the control object as traversing through aparticular level of the presentation tree, sensing a movement of a firstsub-object of the control object in the three dimensional sensor spaceand interpreting the movement of the first sub-object as selecting adifferent level in the presentation tree. It further relates tosubsequently interpreting the movement of the control object astraversing through the different level of the presentation tree, sensinga movement of a second sub-object of the control object in the threedimensional sensor space, interpreting the movement of the secondsub-object as selecting a different presentation layout from a currentpresentation layout of the presentation tree and subsequently presentingthe presentation tree in the different presentation layout.

The technology further relates to automatically determining a control toa virtual control by a control object in a three dimensional sensorspace by distinguishing the control object and a sub-object of thecontrol object by sensing a location of the control object in a threedimensional sensor space, determining whether the control object engagesthe virtual control based on the location of the control object, sensinga movement of the sub-object of the control object in the threedimensional sensor space and interpreting the movement of the sub-objectas a gesture controlling the virtual control if the control objectengages the virtual control.

The technology disclosed also relates to automatically determining acontrol to a virtual control by a control object in a three dimensionalsensor space by sensing a location of the control object in the threedimensional sensor space, determining whether the control object engagesthe virtual control based on the location of the control object, sensingorientation of the control object, defining a control plane tangentialto a surface of the control object and interpreting a direction of thecontrol plane as a gesture controlling the virtual control if thecontrol object engages the virtual control.

The technology disclosed further relates to automatically interpreting agesture of a control object in a three dimensional space relative to oneor more objects depicted in a display by sensing a speed of a movementof the control object moving through the three dimensional sensor space,interpreting the movement as a path on the display if the speed of themovement exceeds a pre-determined threshold and duplicating one or moreof the objects in the display that intersect the interpreted path.

Other aspects and advantages of the present invention can be seen onreview of the drawings, the detailed description and the claims, whichfollow.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to like partsthroughout the different views. Also, the drawings are not necessarilyto scale, with an emphasis instead generally being placed uponillustrating the principles of the technology disclosed. In thefollowing description, various implementations of the technologydisclosed are described with reference to the following drawings, inwhich:

FIG. 1 illustrates an exemplary motion-capture system in accordance withimplementations of the technology disclosed.

FIG. 2 illustrates an exemplary computer system for image processing,analysis, and display in accordance with implementations of thetechnology disclosed.

FIG. 3 shows definition of a control plane with respect to a controlobject according to one implementation of the technology disclosed.

FIG. 4A illustrates a control plane that is more normal to a controlobject's trajectory.

FIG. 4B depicts a control plane that is more parallel to a controlobject's trajectory.

FIG. 5 is one implementation of gesturally controlling an electronicbook.

FIG. 6 illustrates a flow flowing horizontally leftward in a display,and a control plane more parallel to the flow.

FIG. 7 depicts a flow flowing horizontally leftward in a display, and acontrol plane more normal to the flow.

FIG. 8 illustrates a flow having vectors of various directions andmagnitudes at different locations in a display.

FIG. 9 illustrates one implementation of gesturally controlling a spherein a display.

FIG. 10 shows one implementation of gesturally controlling a photo albumicon in a display.

FIG. 11 is one implementation of automatically interpreting gestures ofa control object by distinguishing the control object and a sub-objectof the control object.

FIG. 12 illustrates one implementation of automatically interpretinggestures of a control object to navigate a multi-layer presentation treepresented across a display.

FIGS. 13A-B depict one implementation of gesturally controlling virtualcontrols in a display.

FIG. 14 shows one implementation of gesturally duplicating objects in adisplay.

FIG. 15 is a representative method of automatically interpreting agesture of a control object in a three dimensional sensor space.

FIG. 16 illustrates a method of automatically interpreting a gesture ofa control object in a three dimensional sensor space relative to a flowdepicted in a display.

FIG. 17 shows a flowchart of navigating a multi-layer presentation treeusing gestures of a control object in a three dimensional sensor spaceby distinguishing between the control object and a sub-object of thecontrol object.

FIG. 18 depicts a representative method of navigating a multi-layerpresentation tree using gestures of a control object in a threedimensional sensor space by distinguishing between the control objectand one or more sub-objects of the control object.

FIG. 19 shows a method of automatically determining a control to avirtual control by a control object in a three dimensional sensor spaceby distinguishing the control object and a sub-object of the controlobject.

FIG. 20 is a flowchart of automatically determining a control to avirtual control by a control object in a three dimensional sensor space.

FIG. 21 depicts a representative method of automatically interpreting agesture of a control object in a three dimensional space relative to oneor more objects depicted in a display.

DESCRIPTION

A user can interact with a device incorporating a 3D sensor such asdescribed in U.S. Prov. App. No. 61/816,487 and U.S. Prov. App. No.61/872,538 by using gestures in a 3D sensor space monitored by the 3Dsensor. Interacting with the device often requires the control object(e.g., a hand) exiting the 3D sensor space (a “resetting” gesture) tospecify a control (or engagement of a control) of the device. Thetechnology disclosed relates to methods for interpreting gestures of acontrol object in a 3D sensor space, without requiring the controlobject exiting the 3D sensor space. The method can be implemented by acomputing device incorporating a 3D sensor as described in U.S. Prov.App. No. 61/816,487 and U.S. Prov. App. No. 61/872,538. Oneimplementation of underlying technology to which the further technologydisclosed can be applied is illustrated in FIG. 1 .

Motion-Capture System

Motion-capture systems generally include (i) a camera for acquiringimages of an object; (ii) a computer for processing the images toidentify and characterize the object; and (iii) a computer display fordisplaying information related to the identified/characterized object.Referring first to FIG. 1 , which illustrates an exemplarymotion-capture system 100 including any number of cameras 102, 104coupled to an image analysis, motion capture, and control system 106(The system 106 is hereinafter variably referred to as the “imageanalysis and motion capture system,” the “image analysis system,” the“motion capture system,” “the gesture recognition system,” the “controland image-processing system,” the “control system,” or the“image-processing system,” depending on which functionality of thesystem is being discussed.).

Cameras 102, 104 provide digital image data to the image analysis,motion capture, and control system 106, which analyzes the image data todetermine the three-dimensional (3D) position, orientation, and/ormotion of the object 114 the field of view of the cameras 102, 104.Cameras 102, 104 can be any type of cameras, including cameras sensitiveacross the visible spectrum or, more typically, with enhancedsensitivity to a confined wavelength band (e.g., the infrared (IR) orultraviolet bands); more generally, the term “camera” herein refers toany device (or combination of devices) capable of capturing an image ofan object and representing that image in the form of digital data. Whileillustrated using an example of a two camera implementation, otherimplementations are readily achievable using different numbers ofcameras or non-camera light sensitive image sensors or combinationsthereof. For example, line sensors or line cameras rather thanconventional devices that capture a two-dimensional (2D) image can beemployed. Further, the term “light” is used generally to connote anyelectromagnetic radiation, which may or may not be within the visiblespectrum, and can be broadband (e.g., white light) or narrowband (e.g.,a single wavelength or narrow band of wavelengths).

Cameras 102, 104 are preferably capable of capturing video images (i.e.,successive image frames at a constant rate of at least 15 frames persecond); although no particular frame rate is required. The capabilitiesof cameras 102, 104 are not critical to the technology disclosed, andthe cameras can vary as to frame rate, image resolution (e.g., pixelsper image), color or intensity resolution (e.g., number of bits ofintensity data per pixel), focal length of lenses, depth of field, etc.In general, for a particular application, any cameras capable offocusing on objects within a spatial volume of interest can be used. Forinstance, to capture motion of the hand of an otherwise stationaryperson, the volume of interest can be defined as a cube approximatelyone meter on a side. To capture motion of a running person, the volumeof interest might have dimensions of tens of meters in order to observeseveral strides.

Cameras 102, 104 can be oriented in any convenient manner. In oneimplementation, the optical axes of the cameras 102, 104 are parallel,but this is not required. As described below, each of the 102, 104 canbe used to define a “vantage point” from which the object 114 is seen;if the location and view direction associated with each vantage pointare known, the locus of points in space that project onto a particularposition in the cameras' image plane can be determined. In someimplementations, motion capture is reliable only for objects in an areawhere the fields of view of cameras 102, 104; the cameras 102, 104 canbe arranged to provide overlapping fields of view throughout the areawhere motion of interest is expected to occur.

In some implementations, the illustrated system 100 includes one or moresources 108, 110, which can be disposed to either side of cameras 102,104, and are controlled by image analysis and motion capture system 106.In one implementation, the sources 108, 110 are light sources. Forexample, the light sources can be infrared light sources, e.g., infraredlight emitting diodes (LEDs), and cameras 102, 104 can be sensitive toinfrared light. Use of infrared light can allow the motion-capturesystem 100 to operate under a broad range of lighting conditions and canavoid various inconveniences or distractions that can be associated withdirecting visible light into the region where the person is moving.However, a particular wavelength or region of the electromagneticspectrum can be required. In one implementation, filters 120, 122 areplaced in front of cameras 102, 104 to filter out visible light so thatonly infrared light is registered in the images captured by cameras 102,104. In another implementation, the sources 108, 110 are sonic sourcesproviding sonic energy appropriate to one or more sonic sensors (notshown in FIG. 1 for clarity sake) used in conjunction with, or insteadof, cameras 102, 104. The sonic sources transmit sound waves to theuser; with the user either blocking (“sonic shadowing”) or altering thesound waves (“sonic deflections”) that impinge upon her. Such sonicshadows and/or deflections can also be used to detect the user'sgestures and/or provide presence information and/or distance informationusing ranging techniques. In some implementations, the sound waves are,for example, ultrasound, which are not audible to humans.

It should be stressed that the arrangement shown in FIG. 1 isrepresentative and not limiting. For example, lasers or other lightsources can be used instead of LEDs. In implementations that includelaser(s), additional optics (e.g., a lens or diffuser) can be employedto widen the laser beam (and make its field of view similar to that ofthe cameras). Useful arrangements can also include short-angle andwide-angle illuminators for different ranges. Light sources aretypically diffuse rather than specular point sources; for example,packaged LEDs with light-spreading encapsulation are suitable.

In operation, light sources 108, 110 are arranged to illuminate a regionof interest 112 that includes an entire control object or its portion114 (in this example, a hand) that can optionally hold a tool or otherobject of interest. Cameras 102, 104 are oriented toward the region 112to capture video images of the hand 114. In some implementations, theoperation of light sources 108, 110 and cameras 102, 104 is controlledby the image analysis and motion capture system 106, which can be, e.g.,a computer system, control logic implemented in hardware and/or softwareor combinations thereof. Based on the captured images, image analysisand motion capture system 106 determines the position and/or motion ofhand 114.

Motion capture can be improved by enhancing contrast between the objectof interest 114 and background surfaces like surface 116 visible in animage, for example, by means of controlled lighting directed at theobject. For instance, in motion capture system 106 where an object ofinterest 114, such as a person's hand, is significantly closer to thecameras 102 and 104 than the background surface 116, the falloff oflight intensity with distance (1/r² for point like light sources) can beexploited by positioning a light source (or multiple light sources) nearthe camera(s) or other image-capture device(s) and shining that lightonto the object 114. Source light reflected by the nearby object ofinterest 114 can be expected to be much brighter than light reflectedfrom more distant background surface 116, and the more distant thebackground (relative to the object), the more pronounced the effect willbe. Accordingly, a threshold cut off on pixel brightness in the capturedimages can be used to distinguish “object” pixels from “background”pixels. While broadband ambient light sources can be employed, variousimplementations use light having a confined wavelength range and acamera matched to detect such light; for example, an infrared sourcelight can be used with one or more cameras sensitive to infraredfrequencies.

In operation, cameras 102, 104 are oriented toward a region of interest112 in which an object of interest 114 (in this example, a hand) and oneor more background objects 116 can be present. Light sources 108, 110are arranged to illuminate region 112. In some implementations, one ormore of the light sources 108, 110 and one or more of the cameras 102,104 are disposed below the motion to be detected, e.g., in the case ofhand motion, on a table or other surface beneath the spatial regionwhere hand motion occurs. This is an optimal location because the amountof information recorded about the hand is proportional to the number ofpixels it occupies in the camera images, and the hand will occupy morepixels when the camera's angle with respect to the hand's “pointingdirection” is as close to perpendicular as possible. Further, if thecameras 102, 104 are looking up, there is little likelihood of confusionwith background objects (clutter on the user's desk, for example) andother people within the cameras' field of view.

Control and image-processing system 106, which can be, e.g., a computersystem, specialized hardware, or combinations thereof, can control theoperation of light sources 108, 110 and cameras 102, 104 to captureimages of region 112. Based on the captured images, the image-processingsystem 106 determines the position and/or motion of object 114. Forexample, in determining the position of object 114, image-analysissystem 106 can determine which pixels of various images captured bycameras 102, 104 contain portions of object 114. In someimplementations, any pixel in an image can be classified as an “object”pixel or a “background” pixel depending on whether that pixel contains aportion of object 114 or not. With the use of light sources 108, 110,classification of pixels as object or background pixels can be based onthe brightness of the pixel. For example, the distance (r_(O)) betweenan object of interest 114 and cameras 102, 104 is expected to be smallerthan the distance (r_(B)) between background object(s) 116 and cameras102, 104. Because the intensity of light from sources 108, 110 decreasesas 1/r², object 114 will be more brightly lit than background 116, andpixels containing portions of object 114 (i.e., object pixels) will becorrespondingly brighter than pixels containing portions of background116 (i.e., background pixels). For example, if r_(B)/r_(O)=2, thenobject pixels will be approximately four times brighter than backgroundpixels, assuming object 114 and background 116 are similarly reflectiveof the light from sources 108, 110, and further assuming that theoverall illumination of region 112 (at least within the frequency bandcaptured by cameras 102, 104) is dominated by light sources 108, 110.These conditions generally hold for suitable choices of cameras 102,104, light sources 108, 110, filters 120, 122, and objects commonlyencountered. For example, light sources 108, 110 can be infrared LEDscapable of strongly emitting radiation in a narrow frequency band, andfilters 120, 122 can be matched to the frequency band of light sources108, 110. Thus, although a human hand or body, or a heat source or otherobject in the background, can emit some infrared radiation, the responseof cameras 102, 104 can still be dominated by light originating fromsources 108, 110 and reflected by object 114 and/or background 116.

In this arrangement, image-analysis system 106 can quickly andaccurately distinguish object pixels from background pixels by applyinga brightness threshold to each pixel. For example, pixel brightness in aCMOS sensor or similar device can be measured on a scale from 0.0 (dark)to 1.0 (fully saturated), with some number of gradations in betweendepending on the sensor design. The brightness encoded by the camerapixels scales standardly (linearly) with the luminance of the object,typically due to the deposited charge or diode voltages. In someimplementations, light sources 108, 110 are bright enough that reflectedlight from an object at distance r_(O) produces a brightness level of1.0 while an object at distance r_(B)=2r_(O) produces a brightness levelof 0.25. Object pixels can thus be readily distinguished from backgroundpixels based on brightness. Further, edges of the object can also bereadily detected based on differences in brightness between adjacentpixels, allowing the position of the object within each image to bedetermined. Correlating object positions between images from cameras102, 104 allows image-analysis system 106 to determine the location in3D space of object 114, and analyzing sequences of images allowsimage-analysis system 106 to reconstruct 3D motion of object 114 usingmotion algorithms.

In accordance with various implementations of the technology disclosed,the cameras 102, 104 (and typically also the associated image-analysisfunctionality of control and image-processing system 106) are operatedin a low-power mode until an object of interest 114 is detected in theregion of interest 112. For purposes of detecting the entrance of anobject of interest 114 into this region, the system 100 further includesone or more light sensors 118 (e.g., a CCD or CMOS sensor) and/or anassociated imaging optic (e.g., a lens) that monitor the brightness inthe region of interest 112 and detect any change in brightness. Forexample, a single light sensor including, e.g., a photodiode thatprovides an output voltage indicative of (and over a large rangeproportional to) a measured light intensity can be disposed between thetwo cameras 102, 104 and oriented toward the region of interest 112. Theone or more sensors 118 continuously measure one or more environmentalillumination parameters such as the brightness of light received fromthe environment. Under static conditions—which implies the absence ofany motion in the region of interest 112—the brightness will beconstant. If an object enters the region of interest 112, however, thebrightness can abruptly change. For example, a person walking in frontof the sensor(s) 118 can block light coming from an opposing end of theroom, resulting in a sudden decrease in brightness. In other situations,the person can reflect light from a light source in the room onto thesensor, resulting in a sudden increase in measured brightness.

The aperture of the sensor(s) 118 can be sized such that its (or theircollective) field of view overlaps with that of the cameras 102, 104. Insome implementations, the field of view of the sensor(s) 118 issubstantially co-existent with that of the cameras 102, 104 such thatsubstantially all objects entering the camera field of view aredetected. In other implementations, the sensor field of view encompassesand exceeds that of the cameras. This enables the sensor(s) 118 toprovide an early warning if an object of interest approaches the camerafield of view. In yet other implementations, the sensor(s) capture(s)light from only a portion of the camera field of view, such as a smallerarea of interest located in the center of the camera field of view.

The control and image-processing system 106 monitors the output of thesensor(s) 118, and if the measured brightness changes by a set amount(e.g., by 10% or a certain number of candela), it recognizes thepresence of an object of interest in the region of interest 112. Thethreshold change can be set based on the geometric configuration of theregion of interest and the motion-capture system, the general lightingconditions in the area, the sensor noise level, and the expected size,proximity, and reflectivity of the object of interest so as to minimizeboth false positives and false negatives. In some implementations,suitable settings are determined empirically, e.g., by having a personrepeatedly walk into and out of the region of interest 112 and trackingthe sensor output to establish a minimum change in brightness associatedwith the person's entrance into and exit from the region of interest112. Of course, theoretical and empirical threshold-setting methods canalso be used in conjunction. For example, a range of thresholds can bedetermined based on theoretical considerations (e.g., by physicalmodelling, which can include ray tracing, noise estimation, etc.), andthe threshold thereafter fine-tuned within that range based onexperimental observations.

In implementations where the area of interest 112 is illuminated, thesensor(s) 118 will generally, in the absence of an object in this area,only measure scattered light amounting to a small fraction of theillumination light. Once an object enters the illuminated area, however,this object can reflect substantial portions of the light toward thesensor(s) 118, causing an increase in the measured brightness. In someimplementations, the sensor(s) 118 is (or are) used in conjunction withthe light sources 108, 110 to deliberately measure changes in one ormore environmental illumination parameters such as the reflectivity ofthe environment within the wavelength range of the light sources. Thelight sources can blink, and a brightness differential be measuredbetween dark and light periods of the blinking cycle. If no object ispresent in the illuminated region, this yields a baseline reflectivityof the environment. Once an object is in the area of interest 112, thebrightness differential will increase substantially, indicatingincreased reflectivity. (Typically, the signal measured during darkperiods of the blinking cycle, if any, will be largely unaffected,whereas the reflection signal measured during the light period willexperience a significant boost.) Accordingly, the control system 106monitoring the output of the sensor(s) 118 can detect an object in theregion of interest 112 based on a change in one or more environmentalillumination parameters such as environmental reflectivity that exceedsa predetermined threshold (e.g., by 10% or some other relative orabsolute amount). As with changes in brightness, the threshold changecan be set theoretically based on the configuration of the image-capturesystem and the monitored space as well as the expected objects ofinterest, and/or experimentally based on observed changes inreflectivity.

Computer System

FIG. 2 is a simplified block diagram of a computer system 200,implementing all or portions of image analysis and motion capture system106 according to an implementation of the technology disclosed. Imageanalysis and motion capture system 106 can include or consist of anydevice or device component that is capable of capturing and processingimage data. In some implementations, computer system 200 includes aprocessor 206, memory 208, a sensor interface 242, a display 202 (orother presentation mechanism(s), e.g. holographic projection systems,wearable googles or other head mounted displays (HMDs), heads updisplays (HUDs), other visual presentation mechanisms or combinationsthereof, speakers 212, a keyboard 222, and a mouse 232. Memory 208 canbe used to store instructions to be executed by processor 206 as well asinput and/or output data associated with execution of the instructions.In particular, memory 208 contains instructions, conceptuallyillustrated as a group of modules described in greater detail below,that control the operation of processor 206 and its interaction with theother hardware components. An operating system directs the execution oflow-level, basic system functions such as memory allocation, filemanagement and operation of mass storage devices. The operating systemcan be or include a variety of operating systems such as MicrosoftWINDOWS operating system, the Unix operating system, the Linux operatingsystem, the Xenix operating system, the IBM AIX operating system, theHewlett Packard UX operating system, the Novell NETWARE operatingsystem, the Sun Microsystems SOLARIS operating system, the OS/2operating system, the BeOS operating system, the MAC OS operatingsystem, the APACHE operating system, an OPENACTION operating system,iOS, Android or other mobile operating systems, or another operatingsystem platform.

The computing environment can also include otherremovable/non-removable, volatile/nonvolatile computer storage media.For example, a hard disk drive can read or write to non-removable,nonvolatile magnetic media. A magnetic disk drive can read from or writeto a removable, nonvolatile magnetic disk, and an optical disk drive canread from or write to a removable, nonvolatile optical disk such as aCD-ROM or other optical media. Other removable/non-removable,volatile/nonvolatile computer storage media that can be used in theexemplary operating environment include, but are not limited to,magnetic tape cassettes, flash memory cards, digital versatile disks,digital video tape, solid physical arrangement RAM, solid physicalarrangement ROM, and the like. The storage media are typically connectedto the system bus through a removable or non-removable memory interface.

According to some implementations, cameras 102, 104 and/or light sources108, 110 can connect to the computer 200 via a universal serial bus(USB), FireWire, or other cable, or wirelessly via Bluetooth, Wi-Fi,etc. The computer 200 can include a camera interface 242, implemented inhardware (e.g., as part of a USB port) and/or software (e.g., executedby processor 206), that enables communication with the cameras 102, 104and/or light sources 108, 110. The camera interface 242 can include oneor more data ports and associated image buffers for receiving the imageframes from the cameras 102, 104; hardware and/or software signalprocessors to modify the image data (e.g., to reduce noise or reformatdata) prior to providing it as input to a motion-capture or otherimage-processing program; and/or control signal ports for transmitsignals to the cameras 102, 104, e.g., to activate or deactivate thecameras, to control camera settings (frame rate, image quality,sensitivity, etc.), or the like.

Processor 206 can be a general-purpose microprocessor, but depending onimplementation can alternatively be a microcontroller, peripheralintegrated circuit element, a CSIC (customer-specific integratedcircuit), an ASIC (application-specific integrated circuit), a logiccircuit, a digital signal processor, a programmable logic device such asan FPGA (field-programmable gate array), a PLD (programmable logicdevice), a PLA (programmable logic array), an RFID processor, smartchip, or any other device or arrangement of devices that is capable ofimplementing the actions of the processes of the technology disclosed.

Camera and sensor interface 242 can include hardware and/or softwarethat enables communication between computer system 200 and cameras suchas cameras 102, 104 shown in FIG. 1 , as well as associated lightsources such as light sources 108, 110 of FIG. 1 . Thus, for example,camera and sensor interface 242 can include one or more data ports 244,245 to which cameras can be connected, as well as hardware and/orsoftware signal processors to modify data signals received from thecameras (e.g., to reduce noise or reformat data) prior to providing thesignals as inputs to a motion-capture (“mocap”) program 218 executing onprocessor 206. In some implementations, camera and sensor interface 242can also transmit signals to the cameras, e.g., to activate ordeactivate the cameras, to control camera settings (frame rate, imagequality, sensitivity, etc.), or the like. Such signals can betransmitted, e.g., in response to control signals from processor 206,which can in turn be generated in response to user input or otherdetected events.

Camera and sensor interface 242 can also include controllers 243, 246,to which light sources (e.g., light sources 108, 110) can be connected.In some implementations, controllers 243, 246 provide operating currentto the light sources, e.g., in response to instructions from processor206 executing mocap program 218. In other implementations, the lightsources can draw operating current from an external power supply, andcontrollers 243, 246 can generate control signals for the light sources,e.g., instructing the light sources to be turned on or off or changingthe brightness. In some implementations, a single controller can be usedto control multiple light sources.

Instructions defining mocap program 218 are stored in memory 208, andthese instructions, when executed, perform motion-capture analysis onimages supplied from cameras connected to sensor interface 242. In oneimplementation, mocap program 218 includes various modules, such as anobject detection module 228, an image and/or object and path analysismodule 238, and gesture-recognition module 248. Object detection module228 can analyze images (e.g., images captured via sensor interface 242)to detect edges and/or features of an object therein and/or otherinformation about the object's location. Object and path analysis module238 can analyze the object information provided by object detectionmodule 228 to determine the 3D position and/or motion of the object(e.g., a user's hand). Examples of operations that can be implemented incode modules of mocap program 218 are described below.

The memory 208 can further store input and/or output data associatedwith execution of the instructions (including, e.g., input and outputimage data) as well as additional information used by the varioussoftware applications. Memory 208 can store object library 258 that caninclude canonical models of various objects of interest. In someimplementations, an object being modeled can be identified by matchingits shape to a model in object library 258.

Display 202, speakers 212, keyboard 222, and mouse 232 can be used tofacilitate user interaction with computer system 200. In someimplementations, results of motion capture using sensor interface 242and mocap program 218 can be interpreted as user input. For example, auser can perform hand gestures that are analyzed using mocap program218, and the results of this analysis can be interpreted as aninstruction to some other program executing on processor 206 (e.g., aweb browser, word processor, or other application). Thus, by way ofillustration, a user might use upward or downward swiping gestures to“scroll” a webpage currently displayed on display 202, to use rotatinggestures to increase or decrease the volume of audio output fromspeakers 212, and so on.

It will be appreciated that computer system 200 is illustrative and thatvariations and modifications are possible. Computer systems can beimplemented in a variety of form factors, including server systems,desktop systems, laptop systems, tablets, smart phones or personaldigital assistants, wearable devices, e.g., goggles, head mounteddisplays (HMDs), wrist computers, heads up displays (HUDs) for vehicles,and so on. A particular implementation can include other functionalitynot described herein, e.g., wired and/or wireless network interfaces,media playing and/or recording capability, etc. In some implementations,one or more cameras can be built into the computer or other device intowhich the sensor is imbedded rather than being supplied as separatecomponents. Further, an image analyzer can be implemented using only asubset of computer system components (e.g., as a processor executingprogram code, an ASIC, or a fixed-function digital signal processor,with suitable I/O interfaces to receive image data and output analysisresults).

In another example, in some implementations, the cameras 102, 104 areconnected to or integrated with a special-purpose processing unit that,in turn, communicates with a general-purpose computer, e.g., via directmemory access (“DMA”). The processing unit can include one or more imagebuffers for storing the image data read out from the camera sensors, aGPU or other processor and associated memory implementing at least partof the motion-capture algorithm, and a DMA controller. The processingunit can provide processed images or other data derived from the cameraimages to the computer for further processing. In some implementations,the processing unit sends display control signals generated based on thecaptured motion (e.g., of a user's hand) to the computer, and thecomputer uses these control signals to adjust the on-screen display ofdocuments and images that are otherwise unrelated to the camera images(e.g., text documents or maps) by, for example, shifting or rotating theimages.

While computer system 200 is described herein with reference toparticular blocks, it is to be understood that the blocks are definedfor convenience of description and are not intended to imply aparticular physical arrangement of component parts. Further, the blocksneed not correspond to physically distinct components. To the extentthat physically distinct components are used, connections betweencomponents (e.g., for data communication) can be wired and/or wirelessas desired.

With reference to FIGS. 1 and 2 , the user performs a gesture that iscaptured by the cameras 102, 104 as a series of temporally sequentialimages. In other implementations, cameras 102, 104 can capture anyobservable pose or portion of a user. For instance, if a user walks intothe field of view near the cameras 102, 104, cameras 102, 104 cancapture not only the whole body of the user, but the positions of armsand legs relative to the person's core or trunk. These are analyzed bythe mocap 218, which provides input to an electronic device, allowing auser to remotely control the electronic device and/or manipulate virtualobjects, such as prototypes/models, blocks, spheres, or other shapes,buttons, levers, or other controls, in a virtual environment displayedon display 202. The user can perform the gesture using any part of herbody, such as a finger, a hand, or an arm. As part of gesturerecognition or independently, the image analysis and motion capturesystem 106 can determine the shapes and positions of the user's hand in3D space and in real time; see, e.g., U.S. Serial Nos. 61/587,554,13/414,485, 61/724,091, and 13/724,357 filed on Jan. 17, 2012, Mar. 7,2012, Nov. 8, 2012, and Dec. 21, 2012 respectively, the entiredisclosures of which are hereby incorporated by reference. As a result,the image analysis and motion capture system processor 206 may not onlyrecognize gestures for purposes of providing input to the electronicdevice, but can also capture the position and shape of the user's handin consecutive video images in order to characterize the hand gesture in3D space and reproduce it on the display screen 202.

In one implementation, the mocap 218 compares the detected gesture to alibrary of gestures electronically stored as records in a database,which is implemented in the image analysis and motion capture system106, the electronic device, or on an external storage system. (As usedherein, the term “electronically stored” includes storage in volatile ornon-volatile storage, the latter including disks, Flash memory, etc.,and extends to any computationally addressable storage media (including,for example, optical storage).) For example, gestures can be stored asvectors, i.e., mathematically specified spatial trajectories, and thegesture record can have a field specifying the relevant part of theuser's body making the gesture; thus, similar trajectories executed by auser's hand and head can be stored in the database as different gesturesso that an application can interpret them differently. Typically, thetrajectory of a sensed gesture is mathematically compared against thestored trajectories to find a best match, and the gesture is recognizedas corresponding to the located database entry only if the degree ofmatch exceeds a threshold. The vector can be scaled so that, forexample, large and small arcs traced by a user's hand will be recognizedas the same gesture (i.e., corresponding to the same database record)but the gesture recognition module will return both the identity and avalue, reflecting the scaling, for the gesture. The scale can correspondto an actual gesture distance traversed in performance of the gesture,or can be normalized to some canonical distance.

In various implementations, the motion captured in a series of cameraimages is used to compute a corresponding series of output images forpresentation on the display 202. For example, camera images of a movinghand can be translated by the processor 206 into a wire-frame or othergraphical representations of motion of the hand. In any case, the outputimages can be stored in the form of pixel data in a frame buffer, whichcan, but need not be, implemented, in main memory 208. A video displaycontroller reads out the frame buffer to generate a data stream andassociated control signals to output the images to the display 202. Thevideo display controller can be provided along with the processor 206and memory 208 on-board the motherboard of the computer 200, and can beintegrated with the processor 206 or implemented as a co-processor thatmanipulates a separate video memory.

In some implementations, the computer 200 is equipped with a separategraphics or video card that aids with generating the feed of outputimages for the display 202. The video card generally includes agraphical processing unit (“GPU”) and video memory, and is useful, inparticular, for complex and computationally expensive image processingand rendering. The graphics card can implement the frame buffer and thefunctionality of the video display controller (and the on-board videodisplay controller can be disabled). In general, the image-processingand motion-capture functionality of the system 200 can be distributedbetween the GPU and the main processor 206.

Free Space Gesture Interface with Orientation of a Control Object

FIG. 3 shows definition 300 of a control plane 302 with respect to acontrol object 114 according to one implementation of the technologydisclosed. Applying the technology disclosed, the computing deviceautomatically interprets a gesture of a control object 114 in a 3Dsensor space by discerning a control plane 302 of the control object114, according to one implementation. The computing device first sensesa control object such as a user's hand 114 in the 3D sensor space. Thecomputing device then senses an orientation of the control object 114and determines a surface of the control object 114. For example, asurface of a hand 114 can be the palm back of the hand 114. Thecomputing device defines a control plane 302 tangential to the surfaceof the control object 114. For example, the computing device can definea control plane 302 tangential to the palm of a hand 114, as illustratedin FIG. 3 .

The computing device then interprets a gesture in the 3D sensor spacebased on whether the movement of the control object 114 is more normalto the control plane 302 or more parallel to the control plane 302. Insome implementations, the computing device calculates a trajectory (anangular trajectory) of the movement of the control object, anddetermines whether the gesture engages a virtual control based onwhether the trajectory is more normal or more parallel to the controlplane 302.

FIG. 4A illustrates that the control plane 402 is more normal to thecontrol object's trajectory 404, according to definition 400A. Thecontrol plane 402 is more normal to the trajectory 404 when a normalvector of the control plane 402 is within a pre-determined range from atangent vector of the trajectory 402 intersecting the control plane 402.For example, the control plane 402 is more normal to the trajectory 404when the normal vector of the control plane 402 is within +/−10 degreesfrom the tangent vector of the trajectory 404. For example, the controlplane 402 is more normal to the trajectory 404 when the normal vector ofthe control plane 402 is within +/−20 degrees or within +/−30 degreesfrom the tangent vector of the trajectory 404.

FIG. 4B depicts that the control plane 406 is more parallel to thecontrol object's trajectory 408, according to definition 400B. Thecontrol plane 406 is more parallel to the trajectory 408 when thecontrol plane 406 is within a pre-determined range from a tangent vectorof the trajectory 408 intersecting the control plane 406. In oneexample, the control plane 406 is more parallel to the trajectory 408when the control plane 406 is within +/−10 degrees from the tangentvector of the trajectory 408. In another example, the control plane 406is more parallel to the trajectory 408 when the control plane 406 iswithin +/−20 degrees or within +/−30 degrees from the tangent vector ofthe trajectory 408.

In yet another example 500, the computing device displays an electronicbook (e-book) 502 in its display 506 as illustrated in FIG. 5 . Theelectronic book 502 also includes a page flip icon 504, indicating to auser that the user can use a swipe to left gesture in a 3D sensor spacein front of the display 506 to flip to the next page, or use a swipe toright gesture in the 3D sensor space to flip to the previous page.

A user can flip to the next page by moving his hand in the 3D sensorspace from right to left, with the hand's palm oriented vertically. Thecomputing device senses the hand's trajectory moving horizontally fromright to left, senses an orientation of hand, and determines a controlplane of the hand (tangential to the palm) being more normal to thetrajectory, and interprets the gesture as flipping to the next page.Similarly, the user can flip to the previous page by moving his hand inthe 3D sensor space from left to right, with the hand's palm orientedvertically. The computing device senses the hand's trajectory movinghorizontally from left to right, senses an orientation of the hand,determines a control plane of the hand (tangential to the palm of backof the hand) being more normal to the trajectory, and interprets thegesture as flipping to the previous page.

In some implementations, the user can move his hand horizontally withinthe 3D sensor space with the hand's palm facing downward. Since acontrol plane (tangential to the palm) is more parallel to the hand'strajectory, the computing device does not interpret the gesture asflipping pages of the electronic book 502. Thus to flip two pagesforward from the current page, the user can move his hand from right toleft with the palm oriented vertically, move his hand from left to rightwith the palm facing downward, then move his hand from right to leftwith the palm oriented vertically, all within the 3D sensor space. Incontrast, without considering a control plane, such back-and-forthmovement of the control object within the 3D sensor space would onlyflip one page forward, not two pages forward, in the electronic book502.

Free Space Gesture Interface with a Flow Depicted in a Display

Applying the technology disclosed and with reference to FIG. 6 , thecomputing device can automatically interpret a gesture of a controlobject 114 in the 3D sensor space as describe above relative to a flow602 depicted in a display 604, by defining a control plane 608 of thecontrol object 114. FIG. 6 shows a flow 602 flowing horizontallyleftward in a display 604, and a control object's movement in the 3Dsensor space in front of the display 608.

In example 600 shown in FIG. 6 , the flow 602 can illustrate waterflow's direction and velocity in a river, according to oneimplementation. The flow 602 can illustrate air flow's direction andvelocity in a wind tunnel in another implementation. In yet anotherimplementation, the flow 602 can also illustrate flowing particles in atube or the travel of game pieces moving across the display.

The computing device automatically interprets a gesture of a controlobject 114 in the 3D sensor space relative to the flow 602 depicted inthe display 604 by first sensing a movement 610 of the control object114 in the 3D sensor space. The computing device then senses anorientation of the control object 114, defines a surface of the controlobject 114, and defines a control plane tangential to the surface of thecontrol object 114. For example, the computing device can define acontrol plane 608 of a hand 114 as a plane tangential to the palm orback of the hand 114 as described earlier. The computing deviceinterprets the gesture based on whether the control plane 608 and themovement of the control object 114 are more normal or more parallel tothe flow 602 depicted in the display 604. The control plane 608 is morenormal to the trajectory 610 when a normal vector 606 of the controlplane 608 is within a pre-determined range from a tangent vector of thetrajectory 610 intersecting the control plane 608. For example, thecontrol plane 608 is more normal to the trajectory 610 when the normalvector 606 of the control plane 608 is within +/−10 degrees from thetangent vector of the trajectory 610. For example, the control plane 608is more normal to the trajectory 610 when the normal vector 606 of thecontrol plane 608 is within +/−20 degrees or within +/−30 degrees fromthe tangent vector of the trajectory 610.

In some implementations, the computing device calculates athree-dimensional velocity of the movement of the control object 114.The computing device then weights the control object's three-dimensionalvelocity by projecting the three-dimensional velocity onto a directionparallel to the flow direction 602. The percentage amount of thethree-dimensional velocity projected onto the flow direction 602 isequal an inner product of a unit vector of the three-dimensionalvelocity and a unit vector of the flow. For example, if the controlobject is moving vertically in the 3D sensor space, the calculatedvelocity is zero, since the projection of the three-dimensional velocityonto the flow direction is zero.

The computing device also weights the velocity by a projection of anormal vector 606 of the control plane 608 onto a direction parallel tothe flow 602 depicted in the display 604. For example, in FIG. 6 , thenormal vector 606 for the control plane 608 is at 20 degrees from theflow's direction 602. The weighting on the velocity then is equal to aninner product between the normal vector 606 and a unit vector of theflow's direction 602, or is equal to cosine (20 degrees)=0.94. For FIG.6 , the normal vector 606 for the control plane 608 is perpendicular tothe flow's direction 602. Thus the weighting on the velocity has a valueof 0 (i.e., the normal vector 606 has zero projection onto the flow'sdirection 602).

The computing device then determines an amount the gesture engages avirtual control of the flow depicted 602 in the display 604 based on thecalculated velocity. For example, a user may swipe his hand from rightto left with the hand's palm oriented vertically in the 3D sensor space.The computing device senses the movement of the hand (a control object),sense an orientation of the hand, and define a control plane tangentialto the hand's palm or back (as described earlier). Since the normalvector 606 of the control plane 608 is in parallel to the flow 602depicted in the display 604, there is no weighting (adjustment) to thevelocity of the control object's movement 610. The computing device thencan adjust the speed of the flow depicted in the display based on themovement's velocity. For example, the computing device can adjust thespeed of the flow 602 as the same as the speed of the control object'smovement 610. Thus the user can increase or decrease the speed of theflow 602 depicted in the display 604. In contrast, if the user moves hishand from right to left in the 3D sensor space with the hand's palmfacing downward, the weighting on the velocity of the control object'smovement is zero. That is, the calculated velocity of the controlobject's movement is zero. The computing device then determines thatthere is no change to the flow 602 (i.e., there is no engagement to theflow by the gesture).

A visual object depicted in a display can include one or more vectors.Applying the technology disclosed, a user can use a control object in a3D sensor space to interact with the visual object relative to itscorresponding vector. As shown in example 700, in FIG. 7 , the flow 702has a corresponding vector (e.g., its unit vector) pointing leftward inthe horizontal direction. A user can use his hand to interact with theflow relative to the flow's corresponding vector. Since the normalvector 706 of the control plane 708 is normal to the flow 702 depictedin the display 704, there is weighting (adjustment) to the velocity ofthe control object's movement 710.

As another example 800 shown in FIG. 8 , a flow 802 flows in variabledirection and magnitude, as indicated by vectors of various directionsand magnitudes at different locations in the display 804.

Applying the technology disclosed, the computing device senses amovement of the user's hand 114 (a control object) in the 3D sensorspace as illustrated by the movement trajectory 810. In someimplementations, the computing device determines a portion of the flow802 engaged by the user's gesture. For example, the computing device candetermine a path in the display 804 as projected by the user's handmovement and determine the user's hand movement engages a portion of theflow 802 intersecting the path. In other implementations, the computingdevice determines that the user's hand movement engages the whole flow802 depicted in the display 804. The computing device then sensesorientation of the hand, and defines a control plane 808 tangential to asurface (e.g., palm or back) of the hand. The computing devicedetermines an amount of the hand gesture engages the vectors of the flow802 using the normal vector 806 based on whether the control plane andthe hand's movement are more normal or more parallel to the vectors asdescribed earlier.

FIG. 9 depicts a sphere depicted in a display 904. The sphere (e.g., avirtual earth globe model) has an axial vector 902 corresponding to theaxis of the sphere. Applying the technology disclosed, the computingdevice senses a movement of the user's hand (a control object) in the 3Dsensor space as illustrated by the movement trajectory 910. Thecomputing device then senses orientation of the hand, and defines acontrol plane 908 tangential to a surface (e.g., palm or back) of thehand 114. The computing device then determines an amount of the handgesture engages a virtual control of the visual sphere based on whetherthe control plane 908 and the hand's movement are more normal or moreparallel to the axial vector 902. For example, the computing device candetermine that the hand gesture engages the visual sphere if the normalvector 906 of the control plane 908 is perpendicular (e.g., within+/−10, +/−20, or +/−30 degrees) to the axial vector 902 of the visualsphere. The computing device then determines an amount of interaction bythe hand gesture with the visual sphere by a distance or speed of thehand's movement trajectory in a direction perpendicular to the axialvector 902 in the 3D sensor space. Here, a direction perpendicular tothe axial vector 902 is not necessarily parallel to the display 904. Thecomputing device can calculate a rotational angle to rotate the visualsphere proportional to a distance and direction of the hand gesture. Inexample 900, the computing device can rotate the visual sphere (thevirtual earth globe model) from west to east for 10 degrees for every 20cm the user's hand moves from left to right in a horizontal direction,while the hand's palm is oriented vertically. Or the computing devicecan rotate the virtual earth globe model from west to east for 10degrees, if the user places his hand in a region right to the sphere inthe 3D sensor region and moves his hand 20 cm closer to the display 904(while the hand's palm is oriented vertically). The computing device canalso change a rotation speed of the visual sphere by the hand's velocityin a direction perpendicular to the axial vector 902 in the 3D sensorspace.

An icon (or an operating system or an application) can have one or morevectors. FIG. 10 shows an icon 1002 representing a photo album in adisplay 1004. The photo album icon 1002 has two vectors: a downwardvector corresponds to minimizing the photo album icon. A rightwardvector corresponds to selecting the photo album icon (e.g., selectingand displaying content of the photo album).

Applying the technology disclosed, the computing device senses amovement of a user's hand (a control object) in the 3D sensor space asillustrated by the movement trajectory 1010 in FIG. 10 . The computingdevice then senses orientation of the hand, and defines a control plane1008 tangential to a surface (e.g., palm or back) of the hand 114. Usingthe normal vector 1006, the computing device then determines a controlof the photo album icon 1002 by the hand gesture based on the directionof the hand movement in the 3D sensor space relative to the vectorsassociated with the photo album icon 1002. In example 1000, if the handis moving downward (e.g., with the hand's palm facing downward), thecomputing device determines that the hand gesture enables minimizing thephoto album icon 1002. If the hand is moving rightward (e.g., with thehand's palm oriented vertically), the computing device determines thehand gesture enables selecting the photo album icon 1002.

Applying the technology disclosed, engagement gesture can be a gesturethat can be engaged, and/or non-engaged with a virtual control (knob,slider, etc. displayable on a screen or other presentation device),virtual object (river in the above example, bongo drums, virtual fruitto slash, etc., combinations) or non-object (screen scroll, audiovolume, other OS controls, and combinations). A control object (handportion, tool portion, etc., and combinations) can provide controlinformation to a machine using an engagement gesture. An engagementgesture can become engaged with a virtual control (or object ornon-object) by: (i) presence of the control object in the interactionzone; and (ii) (in the technology disclosed) having a palm angle withinan engagement range (e.g., angle of zero or more but less than 90degrees). The control object can disengage with the virtual control (orvirtual object or non-object) by changing the angle of the palm tovertical (90 degrees). Selection from among multiple virtual controls(or virtual objects or non-objects) can be achieved by proximity or bypresence in a zone (i.e., a region of the interactive zone defined inspace).

Free Space Gesture Interface by Distinguishing a Control Object and aSub-Object of the Control Object

Applying the technology disclosed, the computing device can alsoautomatically interpreting gestures of a control object bydistinguishing the control object and a sub-object of the controlobject. A sub-object of a control object is a portion of the controlobject. In the example 1100 shown in FIG. 11 , a control object can be ahand 1104. A sub-object of the control object can be a finger (1102,1106, 1108) of the hand (as illustrated in FIG. 11 ), or the hand's palmor back 1108. By distinguishing the control object and the sub-object(or additional sub-objects of the control object) and notinginteractions or relative positions and/or motions therebetween, thetechnology disclosed can enable user interaction in a limited size ofinteraction space (e.g., the control object is relatively far away fromthe intended target, or in a small space like aircraft cabins).

The computing device can interpret a user's gestures in a 3D sensorspace as navigating a multi-layer presentation tree such as a menu (asillustrated in FIG. 12 ) or a file list by distinguishing between acontrol object (of the user) and a sub-object of the control object.

The computing device first senses a movement of the control object inthe 3D sensor space, and interprets the movement of the control objectas scrolling through a particular level 1202 of the multi-layerpresentation tree 1200. For example, the computing devices senses alocation of a hand 1104 (a control object) of a user in the 3D sensorspace, and determines that the location corresponds to the level Bincluding menu items B1, B2 . . . , and B5 of the menu illustratedabove. As the hand moves up and down in the 3D sensor space, thecomputing devices interprets the gesture as moving up and down in thelevel B of the menu.

The computing device senses a movement of a finger (a sub-object such as1102, 1106, 1108, 1110) of the hand 1104, and interprets the movement(e.g., stretching, pointing) of the finger as selecting a differentlevel in the menu. For example, the user moves his hand to a location inthe 3D sensor space corresponding to the menu item B3, and points afinger at the menu item B3. The computing device senses the movement ofthe hand and the finger, and interprets the finger gesture as selectinga different level B3 including menu items B3 a and B3 b. Afterwards, theuser uses his hand to move up and down in the 3D sensor space, and thecomputing device interprets the hand gesture as scrolling through (upand down) the different level B3. The user can also stretch and rotatehis finger in the 3D sensor space to move up a hierarchical level in themenu. The computing device senses the rotating finger and interprets thegesture as selecting the level B including menu items B1 to B5.Afterwards, the user users his hand to move up down in the 3D sensorspace, and the computing device interprets the hand gesture as scrollingup and down the menu level B.

The computing device can also sense a movement of another sub-object ofthe control object to perform additional controls of the menu. Forexample, the user can rotate his hand to change between differentpresentation layouts (e.g., a list view, an icon view, a list view withshort descriptions, etc.) of the menu. The computing device senses thehand's back (a sub-object) and a rotational movement of the back. Inresponse to the rotational gesture, the computing device changes thepresentation layout of the menu (e.g., by cycling through differentpresentation layouts of the menu). That is, the user can use one controlobject (hand) and two attributes of the control object (finger movementand hand rotation) to navigate and change presentation layout of a userinterface (the multi-layer presentation tree).

Applying the technology disclosed, more than one hand attribute(velocity vector, palm normal, curvature of a finger, rotation matrix ofhand, etc., and combinations) in the 3D sensor space can be mapped tomore than one context hierarchy at the same time by the computing devicewith the 3D sensor. This enables the user to traverse multiple pathsthrough a set of menus at the same time. Examples include (i) traversingthe menus with one hand and (ii) traversing menu paths with more thanone hand. For example, a user can use one hand to change channel and theother hand to set volume at the same time. For example, a user canchange channel by pushing with one hand, while turning down the volumeby rotation motion of a finger on the one hand.

The computing device can determine a control of a virtual control by acontrol object (e.g., a hand) by distinguishing the control object and asub-object of the control object. In some implementations, the computingdevice determines whether the control object engages the virtual controland interprets movement of the sub-object as gestures controlling thevirtual control, if the control object engages the virtual control.

A virtual control can present to a user a control function of thecomputing device without a physical switch or dial. In example 1300A,the computing device can present a mute control function with agraphical icon 1302 in the computing device's display 1304, asillustrated in FIG. 13A.

The mute control function has a controllable parameter in it's ON or OFFstate. A user controls the mute control function by moving his hand infront of the mute control icon 1302, and changes the state of the mutecontrol function (from ON to OFF, or from OFF to ON) by stretching afinger of his hand.

The computing device automatically determines the user's control of themute control function by first sensing a location of the user's hand (acontrol object) in the 3D sensor space. The computing device determineswhether the hand engages the mute control function based the location ofthe hand. If the user's hand is in front of the mute control icon 1302in the 3D sensor space, the computing device determines that the handengages the mute control function. The computing device senses amovement of a finger (a sub-object) of the hand. If the computing devicehas determined that the hand has engaged the mute control function, thecomputing device then interprets the movement of the figure (e.g.,finger stretching) as a gesture controlling the mute control function,e.g., turning on or off the mute control function.

FIG. 13B shows another example 1300B of a virtual control. FIG. 13Bdepicts a channel control icon 1308 in the display 1306 of the computingdevice, representing a channel control function. The channel controlfunction allows a user to move up or down of a list of channels (e.g.,TV channels, radio channels), or a list of items, such as a play list ofsongs or videos. The user controls the channel control function bymoving his hand in front of the channel control icon 1308 (in the 3Dsensor space), and moving up the list of channels by rotating his handclockwise, or moving down the list of channels by rotating his handcounter-clockwise.

The computing device automatically determines the user's control of thechannel control function by first sensing a location of the user's hand(a control object) in the 3D sensor space. The computing devicesdetermines that the hand engages the channel control function if thehand is in front of the channel control icon 1308 in the 3D sensorspace. The computing device then senses an orientation of the hand anddefines a surface of the hand such as the hand's palm or back. Thecomputing device defines a control plane tangential to the hand's palm(or back) as described earlier. If the computing device has determinedthat the hand has engaged the channel control function, the computingdevice then interprets the direction (or rotation) of the control planeas a gesture controlling the channel control function. The computingdevice interprets a clockwise rotation of the control plane as moving upthe list of channels, and interprets a counter-clockwise rotation of thecontrol plane as moving down the list of channels.

Duplicating Visual Objects with Free Space Gestures

Applying the technology disclosed, the computing device canautomatically interpret gestures in the 3D sensor space relative toobjects depicted in the display of the computing device. In someimplementations, the computing device can interpret a slicing gesture inthe 3D sensor space as duplicating objects 1402, 1404, 1406 depicted inthe display 1408. In example 1400 illustrated in FIG. 14 , a user useshis hand 1414 and quickly moves his hand 1414 through the 3D sensorspace in front of the computing device's display 1408. The hand'smovement 1412 is mapped to a path 1410 on the display 1408 thatintersects objects B and C. Objects intersecting the path 1410 (B and C)are duplicated. Objects that are not intersected by the path 1410 (e.g.,A) are not duplicated by the hand gestures.

The computing device automatically interprets the hand gestureillustrated in FIG. 14 by first sensing a speed of a movement of thehand (a control object) moving through the 3D sensor space. Thecomputing device interprets the movement 1412 as a path 1410 on thedisplay 1408 if the speed of the movement exceeds a pre-determinedthreshold (e.g., 20 or 30 or 40 cm per second). If the hand is onlyhovering in front of display 1408 with a slower speed (less than thethreshold), then the movement 1412 is not interpreted by the computingdevice as a path on the display 1408. The computing device thenduplicates one or more objects 1402, 1404, 1406 that intersect theinterpreted path 1410 on the display 1408. The computing device mayduplicates objects 1402, 1404, 1406 intersecting the path 1410 if theobjects have been previously selected.

As described in U.S. Prov. App. No. 61/816,487, a user can train thecomputing device the gestures performed in the 3D sensor space describedherein. The output of feedback of the training can be displayed to theuser to exam the integrity of the training process.

FIG. 15 is a representative method 1500 of automatically interpreting agesture of a control object in a three dimensional sensor space.Flowchart 1500 can be implemented by one or more processors configuredto receive or retrieve information, process the information, storeresults, and transmit the results. Other implementations may perform theactions in different orders and/or with different, varying, alternative,modified, fewer or additional actions than those illustrated in FIG. 15. Multiple actions can be combined in some implementations. Forconvenience, this flowchart is described with reference to the systemthat carries out a method. The system is not necessarily part of themethod.

At action 1502, a movement of the control object is sensed in the threedimensional sensor space. In one implementation, the control object is ahand. In another implementation, the control plane is tangential to apalm or back of the hand.

At action 1512, an orientation of the control object is sensed.

At action 1522, a control plane is defined tangential to a surface ofthe control object.

At action 1532, the gesture is interpreted based on whether the movementof the control object is more normal to the control plane or moreparallel to the control plane. In some implementations, the gesture isinterpreted based on whether the movement of the control object is morenormal to the control plane or more parallel to the control plane. Suchimplementations include calculating an angular trajectory of themovement of the control object and determining whether the gestureengages a virtual control based on whether the trajectory is more normalor more parallel to the control plane.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 16 illustrates a method 1600 of automatically interpreting agesture of a control object in a three dimensional sensor space relativeto a flow depicted in a display. Flowchart 1600 can be implemented byone or more processors configured to receive or retrieve information,process the information, store results, and transmit the results. Otherimplementations may perform the actions in different orders and/or withdifferent, varying, alternative, modified, fewer or additional actionsthan those illustrated in FIG. 16 . Multiple actions can be combined insome implementations. For convenience, this flowchart is described withreference to the system that carries out a method. The system is notnecessarily part of the method.

At action 1602, a movement of the control object is senses in the threedimensional sensor space. In another implementation, the control planeis tangential to a palm or back of the hand.

At action 1612, an orientation of the control object is sensed.

At action 1622, a control plane is defined tangential to a surface ofthe control object.

At action 1632, the gesture is interpreted based on whether the controlplane and the movement of the control object are more normal or moreparallel to the flow depicted in the display. In one implementation, thecontrol object is a hand.

In some implementations, the gesture is interpreted based on whether thecontrol plane and the movement of the control object are more normal ormore parallel to the flow depicted in the display. Such implementationsinclude calculating a velocity and weighted velocity of the movement ofthe control object, the weighted velocity being weighted by a projectionof the velocity onto a direction parallel to the flow depicted in thedisplay and by a projection of a normal vector of the control plane ontothe direction parallel to the flow depicted in the display anddetermining an amount the gesture engages a virtual control of the flowdepicted in the display based on the calculated velocity.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 17 shows a flowchart 1700 of navigating a multi-layer presentationtree using gestures of a control object in a three dimensional sensorspace by distinguishing between the control object and a sub-object ofthe control object. Flowchart 1700 can be implemented by one or moreprocessors configured to receive or retrieve information, process theinformation, store results, and transmit the results. Otherimplementations may perform the actions in different orders and/or withdifferent, varying, alternative, modified, fewer or additional actionsthan those illustrated in FIG. 17 . Multiple actions can be combined insome implementations. For convenience, this flowchart is described withreference to the system that carries out a method. The system is notnecessarily part of the method.

At action 1702, a movement of the control object is sensed in the threedimensional sensor space. In some implementations, the control object isa hand and the sub-object is a finger of the hand. In otherimplementations, the sub-object is a portion of the control object.

At action 1712, the movement of the control object is interpreted asscrolling through a particular level of the multi-layer presentationtree (menu, file list).

At action 1722, a movement of the sub-object is sensed in the threedimensional sensor space and the movement of the sub-object isinterpreted as selecting a different level in the multi-layerpresentation tree. In one implementation, the different level is eithera deeper or higher level from the particular level. In otherimplementations, the movement of the control object is subsequentlyinterpreted as scrolling through the different level of the multi-layerpresentation tree.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 18 depicts a representative method 1800 of navigating a multi-layerpresentation tree using gestures of a control object in a threedimensional sensor space by distinguishing between the control objectand one or more sub-objects of the control object. Flowchart 1800 can beimplemented by one or more processors configured to receive or retrieveinformation, process the information, store results, and transmit theresults. Other implementations may perform the actions in differentorders and/or with different, varying, alternative, modified, fewer oradditional actions than those illustrated in FIG. 18 . Multiple actionscan be combined in some implementations. For convenience, this flowchartis described with reference to the system that carries out a method. Thesystem is not necessarily part of the method.

At action 1802, a movement of the control object is sensed in the threedimensional sensor space. In some implementations, the control object isa hand. In other implementations, the first and second sub-objects ofthe control objects are a finger, a palm, or a back of the hand.

At action 1812, the movement of the control object is interpreted astraversing through a particular level of the presentation tree.

At action 1822, a movement of a first sub-object of the control objectis sensed in the three dimensional sensor space and the movement of thefirst sub-object is interpreted as selecting a different level in thepresentation tree. In one implementation, the different level is eitherdeeper or higher level from the particular level. In otherimplementations, the movement of the control object is subsequentlyinterpreted as traversing through the different level of thepresentation tree.

At action 1832, a movement of a second sub-object of the control objectis sensed in the three dimensional sensor space and the movement of thesecond sub-object is interpreted as selecting a different presentationlayout from a current presentation layout of the presentation tree. Inother implementations, the presentation tree is subsequently presentedin the different presentation layout.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 19 shows a method 1900 of automatically determining a control to avirtual control by a control object in a three dimensional sensor spaceby distinguishing the control object and a sub-object of the controlobject. Flowchart 1900 can be implemented by one or more processorsconfigured to receive or retrieve information, process the information,store results, and transmit the results. Other implementations mayperform the actions in different orders and/or with different, varying,alternative, modified, fewer or additional actions than thoseillustrated in FIG. 19 . Multiple actions can be combined in someimplementations. For convenience, this flowchart is described withreference to the system that carries out a method. The system is notnecessarily part of the method.

At action 1902, a location of the control object is sensed in a threedimensional sensor space. In one implementation, the control object is ahand.

At action 1912, a determination is made whether the control objectengages the virtual control based on the location of the control object.

At action 1922, a movement of the sub-object of the control object issensed in the three dimensional sensor space. In one implementation, thesub-object is a finger of the hand.

At action 1932, the movement of the sub-object is interpreted as agesture controlling the virtual control if the control object engagesthe virtual control. In other implementations, the movement of thesub-object is interpreted as a gesture controlling the virtual control.Such implementations include identifying a controllable parameter of thevirtual control and selecting or adjusting the controllable parameterbased on the movement of the sub-object.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 20 is a flowchart 2000 of automatically determining a control to avirtual control by a control object in a three dimensional sensor space.Flowchart 2000 can be implemented by one or more processors configuredto receive or retrieve information, process the information, storeresults, and transmit the results. Other implementations may perform theactions in different orders and/or with different, varying, alternative,modified, fewer or additional actions than those illustrated in FIG. 20. Multiple actions can be combined in some implementations. Forconvenience, this flowchart is described with reference to the systemthat carries out a method. The system is not necessarily part of themethod.

At action 2002, a location of the control object is sensed in a threedimensional sensor space. In one implementation, the control object is ahand.

At action 2012, a determination is made whether the control objectengages the virtual control based on the location of the control object.

At action 2022, an orientation of the control object is sensed.

At action 2032, a control plane is defined tangential to a surface ofthe control object. In one implementation, the surface is tangential toa palm or a back of the hand.

At action 2042, a direction of the control plane is interpreted as agesture controlling the virtual control if the control object engagesthe virtual control. In some implementations, a direction of the controlplane is interpreted as a gesture controlling the virtual control. Suchimplementations include identifying a controllable parameter of thevirtual control and selecting or adjusting the controllable parameterbased on the direction of the control plane.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

FIG. 21 depicts a representative method 2100 of automaticallyinterpreting a gesture of a control object in a three dimensional spacerelative to one or more objects depicted in a display. Flowchart 2100can be implemented by one or more processors configured to receive orretrieve information, process the information, store results, andtransmit the results. Other implementations may perform the actions indifferent orders and/or with different, varying, alternative, modified,fewer or additional actions than those illustrated in FIG. 21 . Multipleactions can be combined in some implementations. For convenience, thisflowchart is described with reference to the system that carries out amethod. The system is not necessarily part of the method.

At action 2102, a speed of a movement of the control object movingthrough the three dimensional sensor space is sensed.

At action 2112, the movement is interpreted as a path on the display ifthe speed of the movement exceeds a pre-determined threshold.

At action 2122, one or more of the objects that intersect theinterpreted path are duplicated in the display. In some implementations,the objects are pre-selected.

This method and other implementations of the technology disclosed caninclude one or more of the following features and/or features describedin connection with additional methods disclosed. Other implementationscan include a non-transitory computer readable storage medium storinginstructions executable by a processor to perform any of the methodsdescribed above. Yet another implementation can include a systemincluding memory and one or more processors operable to executeinstructions, stored in the memory, to perform any of the methodsdescribed above.

The terms and expressions employed herein are used as terms andexpressions of description and not of limitation, and there is nointention, in the use of such terms and expressions, of excluding anyequivalents of the features shown and described or portions thereof. Inaddition, having described certain implementations of the technologydisclosed, it will be apparent to those of ordinary skill in the artthat other implementations incorporating the concepts disclosed hereincan be used without departing from the spirit and scope of thetechnology disclosed. Accordingly, the described implementations are tobe considered in all respects as only illustrative and not restrictive.

What is claimed is:
 1. A method of manipulating one or more objectsdepicted in a display based on interpreting a movement of a controlobject, the method including: interpreting, by a computing device, amovement of the control object through a 3D sensor space sensed using a3D sensor; presenting the interpreted movement as a path on a display;and duplicating a display object, of one or more objects depicted in thedisplay, that intersects the path on the display.
 2. The method of claim1, wherein the duplicated display object is pre-selected before themovement that is interpreted as the path is sensed.
 3. The method ofclaim 1, wherein the movement is interpreted and presented as the pathon the display when a speed of the movement exceeds a pre-determinedthreshold.
 4. The method of claim 3, wherein the pre-determinedthreshold is 20 cm per second.
 5. The method of claim 3, wherein thepre-determined threshold is 30 cm per second.
 6. The method of claim 3,wherein the pre-determined threshold is 40 cm per second.
 7. The methodof claim 1, wherein the duplicating of the display object onlyduplicates display objects, of the one or more objects depicted in thedisplay, that intersect the path.
 8. The method of claim 1, wherein thepath, as interpreted, intersects only two display objects of the one ormore objects depicted in the display, and wherein the duplicatingduplicates the only two display objects that are intersected by the pathas interpreted.
 9. The method of claim 1, wherein the control object isa hand, and the movement is a gesture of the hand.
 10. The method ofclaim 9, wherein the hand includes a detected sub-object that is aportion of the hand, and wherein the method further includes: processinga sensed movement of the sub-object and interpreting the sensed movementof the sensed sub-object as a gesture interacting with a multi-layerpresentation tree.
 11. The method of claim 10, wherein the sub-object isa finger of the hand.
 12. The method of claim 1, wherein the path on thedisplay is interpreted according to the sensed movement of the controlobject, such that the display object and the duplicated display objectare simultaneously displayed.
 13. The method of claim 1, wherein the 3Dsensor comprises one or more cameras that capture images of the 3Dsensor space.
 14. The method of claim 13, wherein the 3D sensor space isilluminated by one or more infrared LEDs, and the cameras are sensitiveto infrared light.
 15. The method of claim 1, wherein the 3D sensorspace is provided with ultrasound waves from sonic sources.
 16. Themethod of claim 15, wherein the 3D sensor comprises one or more sonicsensors that capture sonic shadows or deflections in the 3D sensor spaceof the ultrasound waves.
 17. A non-transitory computer-readablerecording medium having computer instructions recorded thereon, thecomputer instructions, when executed by one or more processors, causethe one or more processors to perform the method of claim
 1. 18. Asystem comprising a memory storing computer instructions and one or moreprocessors, the computer instructions, when executed by the one or moreprocessors, cause the one or more processors to perform the method ofclaim 1.